Skip to content

Add session-aware prefix caching and /v1/stats endpoint - #199

Merged
stikves merged 5 commits into
apple:mainfrom
stikves:sukru/server-prefix-cache
Aug 27, 2026
Merged

Add session-aware prefix caching and /v1/stats endpoint#199
stikves merged 5 commits into
apple:mainfrom
stikves:sukru/server-prefix-cache

Conversation

@stikves

@stikves stikves commented Aug 26, 2026

Copy link
Copy Markdown
Contributor
  • X-Session-ID header enables KV cache reuse across requests in the same conversation (engine's TokenHistory detects prefix overlap)
  • GET /v1/stats returns JSON metrics: throughput, prefix hit rate, memory
  • Remove unconditional engine.reset() from chat path — let the engine handle prefix detection internally for 10-20x TTFT improvement on multi-turn conversations
  • Completions endpoint retains reset (lm-eval sends unrelated prompts)

- X-Session-ID header enables KV cache reuse across requests in the
  same conversation (engine's TokenHistory detects prefix overlap)
- GET /v1/stats returns JSON metrics: throughput, prefix hit rate, memory
- Remove unconditional engine.reset() from chat path — let the engine
  handle prefix detection internally for 10-20x TTFT improvement on
  multi-turn conversations
- Completions endpoint retains reset (lm-eval sends unrelated prompts)
Single-client usage (no header) now gets prefix caching automatically.
Multi-client callers can still differentiate sessions via the header.
@stikves

stikves commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

Tested locally

Loaded 31944 chars (~10648 tokens) from The Project Gutenberg eBook of Crime and Punishment .txt

=== Test: Prefix Reuse (same session, same context) ===
  Q1 (12.38s): ...
  Q2 (13.23s): ...
  Q3 (13.58s): ...
  Prefix hits gained: 2 (expected ≥2)
  PASS

=== Test: Session Isolation ===
  New misses: 2 (expected ≥2)
  PASS

=== Test: Growing Context (multi-turn conversation) ===
  Turn 1 (3.27s): ...
  Turn 2 (3.26s): ...
  Turn 3 (3.27s): ...
  Turn 4 (3.27s): ...
  Prefix hits gained: 3 (expected ≥3)
  PASS

=== Test: Rapid Fire (20 short requests, sequential) ===
  20 requests in 2.2s (0.11s avg), errors: 0
  PASS

=== Test: Stats Coherence ===
  {'avg_prefill_tok_per_sec': 0, 'prefix_hit_rate': 0.819672131147541, 'prefix_misses': 11, 'total_gen_tokens': 861, 'total_requests': 61, 'avg_decode_tok_per_sec': 7.25221759620278, 'prefix_hits': 50, 'total_prompt_tokens': 77572}
  ✓ total_requests > 0
  ✓ prefix_hits + prefix_misses > 0
  ✓ 0 <= prefix_hit_rate <= 1
  ✓ total_gen_tokens > 0
  ✓ hit_rate matches hits/(hits+misses)
  PASS

==================================================
RESULTS:
  ✓ Prefix Reuse
  ✓ Session Isolation
  ✓ Growing Context
  ✓ Rapid Fire
  ✓ Stats Coherence

ALL PASSED

Unit tests are also passing to verify isolated components.

@stikves
stikves marked this pull request as ready for review August 27, 2026 01:12
@stikves stikves self-assigned this Aug 27, 2026
Compute promptTokens.map { Int32($0) } once per handler and reuse for
both prepareForRequest and recordPromptTokens. Eliminates one unnecessary
heap allocation per request (up to 512KB at 128K context).
@stikves
stikves merged commit 35398f2 into apple:main Aug 27, 2026
3 checks passed
@stikves
stikves deleted the sukru/server-prefix-cache branch August 27, 2026 15:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants